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Abstract 

The recently introduced compressive sensing (CS) framework enables digital signal acquisition 
systems to take advantage of signal structures beyond bandlimitedness. Indeed, the number of 
CS measurements required for stable reconstruction is closer to the order of the signal complexity 
than the Nyquist rate. To date, the CS theory has focused on real-valued measurements, but in 
practice, measurements are mapped to bits from a finite alphabet. Moreover, in many potential 
applications the total number of measurement bits is constrained, which suggests a tradeoff 
between the number of measurements and the number of bits per measurement. We study 
this situation in this paper and show that there exist two distinct regimes of operation that 
correspond to high/low signal-to-noise ratio (SNR). In the measurement compression (MC) 
regime, a high SNR favors acquiring fewer measurements with more bits per measurement; in 
the quantization compression (QC) regime, a low SNR favors acquiring more measurements 
with fewer bits per measurement. A surprise from our analysis and experiments is that in many 
practical applications it is better to operate in the QC regime, even acquiring as few as 1 bit 
per measurement. 



1 Introduction 

The compressive sensing (CS) framework has sparked renewed interest in sampling and signal 
acquisition [1, 2]. The framework can be concisely summarized by three fundamental components: 
i) underdetermined linear measurement systems, i.e., we obtain the measurements 

y = $x + e, (1) 

of the signal x 6 M N , with $ £ ~^ MxN and M <C N , and with measurement error e £ ~R M ; ii) 
structured signal models, such as If -sparse signals, i.e., x G := {x € 1^ : ||aj||o := |supp(sc)| < 
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K}; and Hi) computational reconstruction, one example being the convex program known as Basis 
Pursuit Denoising (BPDN), 

x= min II a? II x s.t. ||y — ^x\U < e, (2) 

that guarantees ||a5 — x||2 < Ce for ||e||2 < e, C a constant, and under certain conditions on <3? [3]. A 
significant body of work has been devoted to the study of each of these components individually, e.g., 
by a) characterizing conditions on <I> that provide robust mappings of sparse signals and designing 
physical sampling systems that satisfy such conditions [3-8]; b) proposing more refined classes of 
highly structured signals [9-11]; and c) providing reconstruction guarantees and fast solvers for 
BPDN and other convex programs [12-15] as well as greedy and first-order algorithms [16-18]. 

CS promises to lessen our sampling burden. The simple consequence of (1) is that, when the 
acquisition of each measurement is "expensive," we benefit by sensing only M values rather than N. 
One example of such a situation is magnetic resonance imaging (MRI) [19]. We seek to minimize 
the amount of time to image a patient; however, each measurement is time-consuming, leading 
to a total acquisition time that is currently on the order of tens of minutes. Another example is 
sampling, where the required Nyquist rate for wideband signals may be prohibitively highy [5, 20]. 
It is possible to design a physical sampling system $ such that y = &x = &(x(t)) where a: is a 
vector of Nyquist-rate samples of a bandlimited signal x(t), i 6 f. In this case, (1) translates to 
low, sub-Nyquist sampling rates, a potential boon for wideband acquisition. 

In practice, three issues may arise during signal acquisition that are not modeled by (1). First, 
the real-valued CS measurements will be mapped to discrete bits via a quantizer. Second, there 
may be noise present on the input signal. Third, we often must limit the total number of measured 
bits 23, i.e., we are constrained by a bit-budget when transmitting or storing the measurements. 
Thus, a more precise model of CS acquisition is 

VQ = Q B (<5>(x + n) + e), (3) 

where the signal noise is denoted by n 6 R N , and Qb ■ R — > 21 is a -B-bit scalar quantization 
function (applied element-wise in (3)) that maps real- valued CS measurements to the discrete 
alphabet 21 with |2l| = 2 B . In this paper we will model n as a random vector with each element 
having variance a\. Since the primary source of measurement noise in a well-designed hardware 
system derives from quantization, we will assume ||e||2 = 0. 1 Since the quantizer is scalar, we can 
write the bit-budget constraint as 

<B = MB. (4) 

Although we will focus on scalar quantization in this paper, alternative quantization techniques 
such as sigma-delta [21] or non-monotonic scalar quantization [22] have also been proposed for 
CS systems, as have many algorithms specialized to CS quantization problems [23-29]. The main 
themes presented here will be generally applicable to these techniques and algorithms as well. 

The fixed bit-budget 23 = MB and the signal noise n impose a competing performance tradeoff 
as a function of M. On the one hand, since B = Q3/M, we can increase the bit-depth as we decrease 
the number of measurements, thereby increasing the precision of each measurement. On the other 
hand, signal noise is amplified due to noise folding as we decrease the number of measurements, 

lr The general trends presented in this paper remain unchanged when ||e||2 > 0. 
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thereby decreasing the precision of each measurement [30]. 2 Thus, we find ourselves in somewhat of 
a conundrum: as we take fewer measurements we can allocate more bits per measurement (good), 
but noise folding increases the risk of wasting these bits on already imprecise measurements (bad). 

We can gain more insight into this conundrum through a back-of-the-envelope calculation of 
the optimal total acquisition error, which comprises the expected mean-squared distortion due to 
a scalar quantizer for Gaussian measurements 0(||x||22 _2B ) and the expected reconstruction error 
due to measurement noise O \ Equating these noise levels to minimize the total mean square 

error (MSE) leads to 



This expression can also be found using classical rate-distortion bounds in terms of the signal-to- 
noise ratio (SNR) [33, 34]. Imposing the fixed bit-budget B = *B/M and rearranging terms, we 
find that the MSE is minimized when 



The term on the left is the logarithm of the SNR of the input signal. For fixed 23 and N, (5) implies 
that there are two operational regimes that correspond roughly to "high" input SNR and "low" 
input SNR. At high input SNR, the MSE is minimized by taking a small number of measurements 
M with large bit-depth; we call this the measurement compression (MC) regime. At low input 
SNR, the MSE is minimized by taking a large number of measurements M with small bit-depth; 
we call this the quantization compression (QC) regime. The exact SNR at which the transition 
between the two regimes occurs is a function of the total bit-budget. A primary contribution of 
this paper is to expose and explore the QC regime. 

In this paper we argue for the distinction between the MC and QC regimes in two ways. First, 
we formalize the back-of-the-envelope calculation in (5) by analyzing the reconstruction MSE that 
results from the combined effects of quantization and signal noise folding. Specifically we provide 
an upper bound on this MSE for an optimal non-uniform scalar quantizer that roughly predicts 
the trends of the optimal bit-depth for different signal noise powers and bit-budgets. Second, we 
provide a suite of simulations for a specific setup frequently encountered in practice: the acquisition 
of sparse signals from uniformly quantized measurements. Surprisingly, at certain practical SNRs, 
our simulations suggest that a 1-bit quantizer (using the reconstruction techniques developed in 
[35]) exhibits better performance than larger bit-depth quantizers. 

Revisiting the example CS applications from above, a CS MRI device should aim to operate 
in the MC regime, since the total data acquisition time is proportional to M. In this case, (5) 
recommends acquiring high SNR measurements and quantizing them finely. In contrast, a low 
SNR wideband sampling system should aim to operate in the QC regime. In this case, (5) rec- 
ommends acquiring low SNR measurements and quantizing them coarsely. Fortunately, by some 
divine Providence, sampling rate and bit-depth enjoy an inverse relationship in practical ADCs; 
specifically, we obtain an exponential increase in sampling rate as the bit-depth is decreased [36]. 
Taking this idea to its logical extreme, it has been shown that it is possible to drive the bit-depth 
down to 1 bit per CS measurement and still guarantee stable signal recovery [35, 37-39]. In this 
case the quantizer is simply a comparator, enabling an extremely high sampling rate. 

2 Roughly speaking, noise folding implies that during reconstruction we lose about 3dB of signal-to-noise ratio 
(SNR) as the number of measurements is halved [31, 32]. 





(5) 
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The remainder of this paper is organized as follows. In Section 2, we provide the necessary CS 
background for our analysis and simulations. In Section 3, we develop a bound on the reconstruction 
error due to quantization and signal noise, expressed in terms of a fixed bit-budget. In Section 4, 
we present a series of numerical simulations that further support our argument. We conclude in 
Section 5 with a discussion on the implications of this work. 

2 Background 
2.1 CS Toolkit 

Before examining the effect of noise and quantization on CS reconstruction performance, we first 
review a few key results and definitions that enable our analysis. 

CS reconstruction can be interpreted as consisting of two steps: first finding the non-zero 
coefficient locations (the support) and then estimating the coefficient values. If we can correctly 
identify the true signal support, then the optimal linear estimate for coefficient values can be 
computed via least squares: 

x\n = ^y, x\ n c = 0, (6) 

where &a denotes the submatrix of $ formed by selecting the columns of <E> according to the index 
set x\q is the corresponding subvector of x, VL C is the complement set to ft, and f denotes 
the Moore-Penrose pseudo-inverse. Indeed, if an oracle were to provide the true support Q, then 
no linear CS reconstruction algorithm can perform better than (6). Thus, reconstruction with 
known signal support is sometimes called oracle- assisted reconstruction [32, 40]. Our analysis will 
be primarily in terms of the performance of this best-case reconstruction algorithm. Furthermore, 
from [22, 35], when there is no noise on the measurements, the reconstruction (6) is also consistent, 
meaning that 

Q B ($x) = Q B (<s>nx\n) = QB($n$hy) = Q{y) = y- 

There is no better nonlinear estimator for the quantized measurements than a consistent estimator. 

Robust reconstruction guarantees will only hold for measurement systems $ that are "well- 
conditioned." For instance, the so-called restricted isometry property (RIP) of a matrix $ has been 
shown to be a sufficient condition for the robust recovery of sparse signals via several algorithms [3, 
16]. The RIP of order K with constant 5 is defined as 

(l-d)\\x\\ 2 2 <\\^xf 2 <(l + d)\\x\\l (7) 

for all x e Y,k- Roughly speaking the RIP ensures that the norm of the measurements is close to 
the norm of the signal for all if-sparse signals. An alternative way of thinking of this is that the 
singular values of any submatrix formed by K or fewer columns of <3? are bounded close to 1 ; hence 
any JT-column submatrix of <I> is close to an isometry. 

The RIP ensures stable oracle-assisted recovery when white noise is added to the measurements. 
Specifically, suppose that z = &x — y, where z is a zero-mean random vector with uncorrelated 
(white) entries, each having variance a\. Furthermore suppose that <3? has the RIP of order K, and 
that x is If -sparse. Then Theorem 4.1 of [32] demonstrates that oracle-assisted reconstruction will 
have expected error 

M<E(||*-2|||)<££ (8) 
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A key component of our analysis below will be understanding the variance of the noise term z that 
arises from the quantized noisy measurements yQ. The expression (8) then gives the intuition that 
the expected reconstruction error behaves on the order of the variance of the error per measurement 

°l 

We will also make use of a result that relates the variance a\ of the signal noise to the variance 
of the measured noise o\ n . If n is white with mean zero and variance cr^, and has orthonormal 
rows, i.e., = jjIm, 3 then it is straightforward to show that the measured noise is also white 
and zero mean and has variance 



Note that the measured noise is only uncorrelated (i.e., white) when M < N; indeed, the condition 
$<J> T = ?jLi M can on iy hold when M < N. 

In [32], the authors combine the results of (8) and (9) to obtain a bound on the oracle-assisted 
reconstruction error due to noise folding. We will take a similar approach, however we will ad- 
ditionally include the effects of quantization. Furthermore, because our quantization error is not 
necessarily uncorrelated, we first generalize (8) to obtain an upper bound on the oracle reconstruc- 
tion error with uncorrelated measurement noise. 



The results of the conventional CS framework above will enable us to analyze scalar quantized 
measurements when the bit-depth is greater than 1. However, CS measurements can be coarsely 
quantized to just 1 bit, representing their signs. These facts preclude 1-bit CS from being analyzed 
within the conventional linear CS framework. Even though meaningful theoretical comparisons 
are difficult to make between 1-bit and conventional CS, it is beneficial to compare their empirical 
performances, since both types of CS can be useful in practice. Thus, we very briefly review the 
key results of the 1-bit CS framework [35, 37]. Formally, 1-bit measurements can be written as 



To reconstruct, we search for a sparse, unit-norm signal x that is consistent with the measurements, 
meaning that A[x) = A(x). We restrict our attention to unit-norm signals, since the scale of the 
signal is lost during the 1-bit quantization process. This problem is generally non-convex, and thus it 
is difficult to design an algorithm that will be guaranteed to find the desired solution. Nonetheless 
several algorithms have been proposed to approximately solve this problem [35, 37-39]; convex 
programs have also been formulated [41]. 

In much the same way that the RIP of <& guarantees stable reconstruction from ^-minimization 
programs [42], the so-called binary e-stable embedding (BeSE) provides a similar robustness for the 
mapping A with consistent algorithms [35]. The property explains that the normalized Hamming 
distance between any two sets of measurements is within e of the normalized angular distance 
between the original signals, for all unit-norm K-sparse signals. It can be shown that, if the elements 
of <E> are drawn from a Gaussian distribution, then $> satisfies the BeSE with high probability, and 
thus CS systems that enable 1-bit quantized measurements exist. 

3 The so-called tight frame condition $<3? T = §jIm is not overly restrictive, since for any RIP matrix T, a matrix 
that has both the same row-space as T and the tight frame condition can be derived from T [32]. 




(9) 



2.2 1-bit CS 



y s = A(x) := sign(<J>a:). 



(10) 
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Our simulations will make use of two 1-bit CS algorithms originally introduced in [35]. Specif- 
ically, we will employ the BIHT and BIHT-^2 algorithms. The former can be thought of as min- 
imizing a one-sided £i-norm and imposing a sparse unit-norm signal model, while the latter can 
be thought of as minimizing a one-sided ^-norm instead. By one-sided norm, we mean that the 
positive elements of a vector are set to zero before the norm is computed. The BIHT algorithm has 
been shown to perform better in low noise scenarios, while the BIHT-^2 algorithm has been shown 
to perform better in high noise scenarios [35]. 

3 Analysis of Quantized CS Systems with Signal Noise 

In this section we derive a new upper bound on the oracle-assisted reconstruction error due to both 
noise and quantization, making the back of the envelope calculation (5) more rigorous. This bound 
enables us to argue that, for a fixed bit-budget 25 = MB, it may be better to quantize to fewer 
bits per measurement B than take fewer measurements M. The following theorem is proved in 
Appendix A. 

Theorem 1. Suppose that uq = Qb{<&{x + n)). Let the signal x € 1^ be sparse with support 
f2 G {1,...,N} and \Q\ = K, where the elements SI are chosen uniformly at random and the 
amplitudes of the non-zero coefficients are drawn according to Xj £ SI ~ M(0,a 2 ). Let the signal 
noise n £ M M be a random, white, zero-mean vector with variance . Furthermore, let the M x N 
matrix $ satisfy the RIP of order K with constant 5, $$ T = jjIm, and M < N. Choose Qb 
to be the optimal scalar quantizer with B > 1 that minimizes the MSE for the distribution of the 
measurements &(x + n). Then for a fixed bit-budget of 25 = MB, the MSE of the oracle- assisted 
reconstruction estimate x satisfies 

■ (I — S (*4«r» + NalB (l + 2-)) + ^ (§ - l) 6, (11) 

where & = max^- \F,(Qb(Qx + &n)iQ.B(&x + Qn)j)\ is the correlation between the quantized 
measurements. 

Each component of the bound (11) is fairly intuitive. The term Ka^.B2~ 2B reflects the error 
due to quantizing the measurements. The term Na^B (2~ 2B + l) reflects both the error due to 
measured signal noise as well as the quantization of that noise. The reconstruction error is effectively 
proportional to these two terms. The final term — l) 6 reflects an additional error due to the 
correlation between the quantized measurements. In many CS scenarios we expect this term to 
be close to zero, and furthermore for large B it has been shown that this term can be accurately 
approximated as zero [43]. Thus, choosing the optimal B primarily comes down to balancing the 
terms inside the parentheses. 

The bound in (11) applies to strictly sparse signals immersed in signal noise. However, it may 
also be of interest to consider so-called compressible signals, i.e., signals that are not strictly sparse 
but that can be reasonably approximated by retaining their K largest magnitude coefficients. For 
such signals, the "tail" part of the signal that we do no expect to recover, i.e., the subset of the 
smallest N — K entries, is also subject to noise folding. Theorem 1 can be extended to handle 
compressible signals by inflating the second term to account for the additional correlation between 
the quantized measurements. The general performance trends will be similar to sparse signals in 
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Figure 1: Upper bound on the oracle-assisted reconstruction error as a function of bit-depth B and ISNR. The term 
inside the parenthesis in the bound (11) was computed. Black dots denote the minimum point on each curve. 

noise; i.e., signals that are "less compressible" will induce the same regime as signals with low input 
SNR. 

The bound in (11) is pessimistic, since we do not take into account the benefits accrued by 
increasing the number of measurements, for instance by improving the RIP constants of <£. Fur- 
thermore, when the quantization error is large enough to dominate the measurement noise, the 
measurement noise terms may not play an active role in the true behavior of the system. Again, 
this is not reflected by the bound. Finally, the bound does not apply to 1-bit quantization or the 
case where M > N. 

To use the bound (11) to support our argument that there are both MC and QC regimes in CS, 
we examine the behavior of the oracle-assisted reconstruction error as a function of the bit-depth 
B (or equivalently the number of measurements M since 25 = MB). Since the solution for the 
optimal B cannot be computed in closed form without resorting to tabulated functions, we evaluate 
the bound over some interesting parameters. The evaluation of the bound is depicted in Figure 1, 
where plots (a)-(d) correspond to input signal-to-noise ratios (ISNRs) of 35dB, 20dB, lOdB, and 
5dB, respectively. We define the input SNR (ISNR) in dB as 

isNR; = ioi ° g '«(S§)- < i2 » 

where E(||aj||l) = Ka 2 x and l(||n|||) = Na 2 n . 

Since we are primarily concerned with the performance trend of (11) as a function of B and the 
ISNR, we make a few simplifications when plotting the bound. First, we only evaluate the term 
inside the parenthesis; this term is proportional to the error on the measurements and does not 
depend on the RIP constant, the sparsity K, or the correlation between the quantization errors. 
Second, by only evaluating the term inside the parenthesis in (11), we do not take into account the 
effect of M on the RIP constants (5 decreases as M increases). The minimum error point in each 
curve is denoted by a solid black dot. 
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The message from Figure 1 is clear. The tradeoff between the number of measurements M and 
bit-depth B empirically follows a convex curve, i.e., the error not only increases when B is too 
small, but the error also increases when B is too large. In other words, more bits per measurement 
is not necessarily optimal. Furthermore, as expected, the minimum reconstruction error occurs 
for smaller B as the ISNR decreases. For the high ISNR of 35dB, the bound is minimized at a 
bit-depth of approximately 7 bits per measurement. The is an example of the MC regime, where 
larger bit-depths and thus lower M yield the best performance. For the low ISNR of lOdB, the 
bound is minimized at a bit-depth of approximately 2 bits per measurement. This is an example 
of the QC regime, where larger bit-depths and thus higher M yield the best performance. 

4 Experiments 

In the previous section we have argued that the QC regime exists by deriving an upper bound 
on the oracle-assisted reconstruction error. In this section we perform a suite of simulations to 
empirically study for which input noise levels and bit-budgets this regime will occur in practical 
systems. Specifically our simulations i) validate the theoretical result in Theorem 1, ii) demonstrate 
the performance achieved in practice when combining quantization and signal noise, and finally Hi) 
prove the existence of the QC regime. A surprising additional result emerges from the simulations: 
when nontrivial signal noise is present, 1-bit CS systems perform competitively with, if not better 
than conventional CS with uniform multibit quantization. 

4.1 Setup 

Our simulations were performed using canonically (identity) sparse signals x. A The signals were 

measured with i.i.d. Gaussian matrices, i.e., y = $>(x + n) where the matrix has elements fy j ''~ ' 
A/"(0, 1/M). The measurements were quantized uniformly with quantization interval A = T2~ B+1 , 
where T is the dynamic range of the quantizer. In all simulations, we chose T = ||$a;||oo to maximize 
the range of the quantizer and ensure that for any noiseless measurement |($x)j — Q,B({^x)i)\ < 
A/2. 

In each trial we drew a new MxN sensing matrix <I> and a new signal x. The non-zero coefficients 
of x were chosen according to a Gaussian distribution, and their positions were chosen at random. 
We additionally added Gaussian noise to x to obtain the desired ISNR. For B > 1, reconstruction of 
the estimate x was performed using the oracle-assisted reconstruction algorithm (8) for Section 4.2 
and BPDN (2) with an oracle value of e = \\y — Qb{h)\\2 for the remaining subsections. For 
B = 1, reconstruction was performed using both the binary iterative hard thresholding (BIHT-^i) 
and BIHT-^2 algorithms; the former generally performs better in lower noise scenarios and the 
latter performs better in higher noise scenarios [35]. We report the reconstruction SNR (RSNR) 



in dB unless otherwise noted. Recall that the number of measurements and bit-depth are 
constrained by *B = MB. We average our results over 100 trials for each parameter tuple 




(13) 



(N, K, <8,B, ISNR). 



4 The results of simulations did not change when the signals were DCT-sparse. 
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(a) ISNR = 35dB, optimal bit-depth 
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(b) ISNR = 20dB, optimal bit-depth = 6 
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(c) ISNR = lOdB, optimal bit-depth = 4 
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(d) ISNR = 5dB, optimal bit-depth = 3 



Figure 2: Oracle-assisted reconstruction error (compare to the analytical upper bound plotted in Figure 1) for 
N = 1000, K — 10, and 23 = 3N. As predicted by (11), the minimum reconstruction error (denoted by black dots) 
is achieved by smaller bit-depths as the ISNR decreases. 



4.2 Oracle-assisted reconstruction 



We begin by validating the message from Theorem 1, i.e., we examine the solution to the oracle- 
assisted reconstruction algorithm to see how the empirical performance relates to the bound (11). 
Our goal is to compare the performance of our simulations to the theory-based plots in Figure 1. 
The experiments were performed as described previously with the oracle-assisted reconstruction 
algorithm. We plot the reconstruction error \\x — for bit-depths between 2 and 12 for a fixed 
bit-budget 23 = 3iV. We compared bit-depths of 2 and higher, since (11) does not hold for lower bit- 
depths. Furthermore, unlike the statement of Theorem 1, recall that we used a uniform quantizer 
and not an optimal quantizer for the Gaussian measurements. Figures 2(a)-(d) depict the results 
for ISNR = 35dB, 20dB, lOdB, and, 5dB, respectively. 

The plots generally follow the same trends as in Figure 1; however the minimum error occurs 
for a slightly higher bit-depth in each case. The plots demonstrate that, as claimed in Section 3, 
the best performance is obtained for smaller bit-depths as the ISNR decreases. 



4.3 Reconstruction performance as a function of 03 

We next explore the performance achieved using practical algorithms instead of oracle-assisted 
reconstruction. The experiments were performed as explained previously, for N = 1000 and K = 
10, bit-depths B = 1,2,4,6,8,10,12, and for bit-budgets *B G [N/2,7N], with the BPDN and 
BIHT algorithms. Figures 3(a)-(d) depict the experiment for the input ISNR = 35, 20, 10, 5dB, 
respectively. 

In the high ISNR regime of 35dB, bit-depths of B = 1, 6, 8, 10, and 12 obtain similar RSNRs of 
around 35dB, while smaller bit-depths result in poorer performance. This is to be expected; since 
when the signal noise is fairly small, we will generally do better by using more bits per measurement. 
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Figure 3: Reconstruction performance as a function of total bits, for different ISNRs. Plots depict RSNR for different 
bit-depths B for different ISNR with parameters N = 1000 and K = 10, and reconstruction via BPDN. The figure 
demonstrates that as the ISNR is decreased, smaller bit-depths achieve better performance. Additionally, 1-bit CS 
techniques perform competitively with or better than BPDN for all ISNRs tested. 



The performance of BIHT in this case is consistent with previous results showing that the 1-bit 
techniques can outperform even 4-bit uniformly quantized CS measurements with BPDN recovery. 
This trend starts to reverse for lower signal ISNRs. Indeed for ISNRs of lOdB and 5dB, we see 
that 2 and 4 bit-depth quantization outperforms larger bit-depths for all budgets. Strikingly, the 
best performance for input SNRs of 20dB, lOdB, and 5dB is achieved by acquiring just 1 bit per 
measurement and reconstructing with the BIHT-^2 algorithm. 

In addition to the simulations presented here, we also performed the similar simulations with 
N = 1000 and K = 60. We found that all of the curves in Figure 3 dropped in SNR by roughly the 
same constant (that depends on K). The relationship between the 1-bit curves and the others was 
about the same for 55 = 2iV and lower. For 03 > 2N, the 1-bit reconstructions still outperformed 
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ISNR (dB) ISNR (dB) ISNR (dB) 

(a) 53 = N (b) 53 = 2N (c) 53 = 5iV 

Figure 4: Maximum RSNR given a fixed bit-budget 23 for parameters iV = 1000, K — 10. The left side of each plot 
corresponds to the QC regime, while the right side corresponds to the MC regime. The solid line (blue) corresponds 
to the number of measurements M, while the dashed line (green) corresponds to the bit-depth B. 



the others; however the performance disparity was not as great as for K = 10. 

These simulations demonstrate two points. First, they verify that the intuition provided by the 
upper bound (11) is indeed correct: for lower ISNRs it is beneficial to choose smaller bit-depths 
B and more measurements M . This validates the distinction between the QC and MC regimes. 
Second, the 1-bit CS setup performs significantly better than the multi-bit setup for low ISNRs 
and is competitive with the multi-bit setup for moderate ISNRs. There are several reasons for 
this. When the quantization error dominates the measurement noise, the reconstruction error is 
primarily due to the quantization error only. This case arises when B is small; i.e., we can likely 
satisfy Qb(x + n) = Qb(x) for increasing values of \ni\ as B decreases. Furthermore, in this case 
consistent reconstruction of the 1-bit algorithms may have an advantage. Consistency could be 
presumably added to multibit reconstruction to improve performance but this is a topic left for 
future research. 

4.4 Reconstruction performance as a function of ISNR 

In this set of experiments, we varied the ISNR between 5dB and 45dB and searched for the (M, B) 
pair that maximized the RSNR, for a fixed bit-budget 53 and parameters N = 1000 and K = 10. 
As demonstrated by the previous experiment, the RSNR will not be the same for each bit-budget. 

Figures 4(a)-(c) depict the results of this experiment for 53 = N, 2N, and 5iV, respectively. 
The left axis and solid line (blue) corresponds to the number of measurements M , while the right 
axis and dashed line (green) corresponds to the bit-depth B. As always, we have that 53 = MB. 
The QC regime is represented on the left side of the plots (low ISNR), while the MC regime is 
represented on the right side of the plots (high ISNR). For example, for a bit-budget of 53 = 27V, 
if the ISNR is 30dB, then we are operating in the MC regime and should set the bit-depth to 
approximately 7, resulting in the measurement ratio of approximately M/N = 0.29. However, for 
the same bit-budget, if the ISNR is 15dB, then we are operating in the QC regime and should set 
the bit-depth to 1, resulting in a measurement ratio of M/N = 2. 

In each plot in Figure 4 there is a sharp transition between optimal bit-depth being high (B > 5) 
and low (B < 2). This transition is centered at the ISNRs 19dB, 23dB, and 38dB, for the bit- 
budgets 53 = N, 2N, and 5iV, respectively. This implies that the transition occurs at higher ISNRs 
for higher bit-budgets. Thus, we infer that, for higher bit-budgets 53, it is better to choose low B, 
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even when the input ISNR is fairly high. The bottom line then is that, for moderate ISNR, the 
MC regime can be assumed when the bit-budget OS is small, while the QC regime can be assumed 
when the bit-budget is large. 



5 Discussion 

In this paper we have studied compressive sensing (CS) systems with scalar quantization when the 
total number of measurement bits is fixed and noise is present on the input signal. Our results 
have demonstrated that in CS, it is sometimes better to reduce the bit-depth than the number of 
measurements. We found that there exist two regimes: in the high-ISNR, MC regime, we should 
compress by reducing the number of measurements; this regime is best suited for applications where 
the acquisition if each measurement is expensive. In the low-ISNR, QC regime, we should compress 
by reducing the number of bits per measurement; this regime is best suited to applications where 
the acquisition of each measurement is cheap, or large bit-depth quantizers are expensive. The key 
to exposing the QC regime was the recognition that there is a tradeoff between amplified input 
signal noise (due to the underdetermined measurement system) and the number of bits that can 
be allocated per measurement under a fixed bit-budget. 

Choosing a low bit-depth quantizer to reduce hardware complexity while driving up the sam- 
pling rate, as is recommended for the QC regime, is not a new idea. Indeed, this same principle 
is the motivational force behind sigma-delta ADCs [44-47] and other non-CS oversampled ADC 
architectures [48-50] . However, the ideas presented here differ significantly from previous oversam- 
pled ADC architectures in the following ways: i) CS is compressive: Small bit-depth CS systems 
are expected to be used in cases where the bit-budget is significantly lower than in a conventional 
oversampled ADC system. The use of sparse signal models enables compression, i.e., a reduction 
in the total number of acquired bits, as opposed to just efficient sampling, ii) CS is non-adaptive: 
As described earlier, CS measurement systems are non-adaptive, meaning they do not depend on 
the input signal. This is true even for the 1-bit CS case. Almost all previous oversampled ADCs 
require some kind of feedback during quantization to produce stable representations. These dif- 
ferences place low bit-depth CS systems in a unique class of their own. In a few words, CS, like 
physics has "plenty of room at the bottom [51]." 

A Proof of Theorem 1 

We first extend the upper bound of Theorem 4.1 in [32] on the oracle-assisted reconstruction error 
to account for correlated measurement noise. 

Lemma 1. Suppose that y = <kx + z, where z G M M is a zero-mean, random vector with covariance 
matrix X = K(zz T ), and that x is K-sparse. Furthermore, suppose that <I> satisfies the RIP of order 
K with constant 5. Then the estimate x provided by the oracle- assisted reconstruction algorithm 
(8) satisfies 




(14) 



where A max (£) is the largest eigenvalue o/E. 



12 



Proof. For a fixed support set fi € {1, . . . , N} with = K, the RIP ensures that &q is full rank, 
and thus the oracle estimate satisfies 



sB|n = aj|n + 4 2 - ( 15 ) 

We seek to estimate E (ll^^Hl)- 

For any K x M matrix A we have that 

E(||Az|||) = E(Tr(Az(Azf)) =E(Tr(Azz T A T )) 
= Tr{AE(zz T )A T ) = Tr(ASA T ) 

= J2 X MZA T ), (16) 

3=1 

where Xj(AT,A T ) denotes the j-th eigenvalue of AT>A T , and (16) follows since AT,A T is a K x K 
matrix. Lemma 8.2 of [32] explains that the eigenvalues of this matrix can be upper bounded as 

A max (,4£,4 T ) < A max (A4 T )A max (£) 

< Smax(A) 2 A max (S), (17) 

where s m ax(^4) denotes the maximum singular value of A. 

Thus, to obtain the final bound, we combine (16) with (17) and substitute A = yielding 



E (H^ll 2 ) < K Smax (c4) 2 A max (£) 

< ^A max (S), (18) 

since we have that Smax^^) 2 < j^s horn Lemma 8.1 of [32]. □ 

We next demonstrate that, by choosing a signal model with random values and supports, the 
noiseless measurements <&x are identically distributed and uncorrelated. 

Lemma 2. Let x € be a sparse signal with support Q G {1, ...,N} and \Q\ = K, where 
the elements £1 are chosen uniformly at random and the amplitudes of the non-zero coefficients are 
drawn according to Xj € ~ A/"(0, a^.). Furthermore, let the M x N matrix <E> satisfy 3><!> T = jjIm- 
Then the vector §x is distributed as a mixture of Gaussians with 

E((*x)i) = 0, E(($x)(*x) T ) = ^C7 2 X I M , (19) 
i.e., the elements (3>£c)j of &x are zero-mean uncorrelated variables. 

Proof. For a fixed support each element ($>x)i is Gaussian distributed with mean zero since 
it is the sum of K zero-mean Gaussian variables. Furthermore, the distribution of (3>a?)j over all 
possible supports is the sum of the distribution for each fixed support, scaled by the probability 
that they occur. Thus, (3>a?)j is a mixture of Gaussians with E(($x)j) = 0. 

To derive the variance of the elements and also show that they are uncorrelated, we first examine 
E{xx T ). The off-diagonal elements are zero, i.e., E(xiXj)i^j = 0, since the elements of x are 
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uncorrelated, by definition. Furthermore, the variance of the diagonal elements can be computed 

as 

E(x?) = a 2 x F(i 6fl) = ^a 2 x , 



since the K non-zero support locations are chosen uniformly, any location j is chosen with proba- 



bility K/N. Thus, E(xx T ) = jt(t x In- We next compute the correlation of the measurements <&x 



to obtain 



E($a;($sc) T ) = <5>E(xx T )$ T 



= ^a 2 x ^ T =^a 2 x I M , (20) 

which concludes the proof. □ 
Proof of Theorem 1. Denote the error between the noiseless ideal measurements and uq by 

z := &x — Qb(&x + $n). (21) 

Our goal is to determine a bound on the variance a 2 , of each element of z. We begin by rewriting 
the norm squared of z as 

z 2 = [^x)i-QB^x + ^n)i)] 2 

< 2[{$x + <S>n)i - Q B {$x + <5>n)i} 2 + 2(<5>n) 2 , (22) 

where the index i denotes individual elements of the respective vector. 

We now seek an upper bound on the expected value of each of the quantities in (22). We begin 
with the second term in (22). From the definition of <J>, we have that the elements of <E>n have 
variance 

4 n = E(($n)?) = ^o- 2 n , (23) 

and furthermore are uncorrelated, as was reviewed in Section 2. 

To bound the first term in (22), we note that the optimal scalar quantizer of rate B for a 
Gaussian variable g with variance a has MSE given by E(g - Q B {g)) 2 = cr 2 2- 2B . Furthermore, 
the MSE of an optimal quantizer of rate B for any variable with variance a 2 is upper bounded by 
that of a Gaussian variable. Our goal is to apply this quantization bound to (Qx + $n)j. Since 
($£c)j and (3>n)j are zero mean and independent of each other, then we immediately have that 
E(^x + ^n) 2 ) = §a 2 x + 

jtf a ni where the first term follows from Lemma 2, and the second term 
follows from (23). Thus, we can bound the first term in (22) as 



E + *n)j - Q B {<bx + $n)i} 2 ) < E + $n)f ) 2 



-IB 



< T7 <&~ 2B + Tttftr™. (24) 



Combining (23) and (24) as in (22) yields 



4 < 2^^2" 2B + 2^^ (l + 2~ 2B ) . (25) 
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We have thus far established an upper bound on the variance u\. of the error Z{ of each mea- 
surement. We next obtain a bound on the eigenvalues of the covariance matrix £ = K(zz T ). The 
off-diagonal elements of £ can be written as 

E(zjZj)i# = E((*as) i (*a5)j))-E((*aj)iQB(*a5 + $n) j ) 

-E(($aj)jQ B (*a; + + E(Q B ($x + $n)iQ B ($a; + $n)j) 

= -E(Q B (*« + *n)iQB(*SB + ^n)j), (26) 

since E((<J?;c)i(3>£c)j)) = by design and, for an optimal scalar quantizer, we have that E,(Qb(&x + 
^ > «)iQs(^ > « + $«-)i) = E(($x)_,-Qs($x + $n)i) [43]. Thus, the matrix £ has of along its diagonal 
and (3 for all other entries. We next apply Gershgorin's circle theorem, which explains that any 
eigenvalue is upper bounded by the diagonal entry plus the sum of the magnitudes of the off-diagonal 
entries of each row of E. Thus, we have 

Amax(E) < u 2 z . + (M — 1)6, (27) 

where & = max^j \E(ziZj)\. 

To obtain the final bound, we combine to (25) with (27) and apply the upper bound in Lemma 1. 
We express the bound with the substitution M = *B/B. □ 
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